Skip to content

[VMware to KVM] Cleanup leftover migrated volumes in case of migration failures#13151

Merged
weizhouapache merged 7 commits into
apache:mainfrom
shapeblue:423-vmw-kvm-cleanup-migrated-disks-failure
Jul 8, 2026
Merged

[VMware to KVM] Cleanup leftover migrated volumes in case of migration failures#13151
weizhouapache merged 7 commits into
apache:mainfrom
shapeblue:423-vmw-kvm-cleanup-migrated-disks-failure

Conversation

@nvazquez

Copy link
Copy Markdown
Contributor

Description

This PR includes a cleanup mechanism for migrated volumes which are leftover in case the VMware to KVM migrations have failed unexpectedly.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@codecov

codecov Bot commented May 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 35.74144% with 169 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.94%. Comparing base (82986f6) to head (ce87ce4).

Files with missing lines Patch % Lines
...urce/wrapper/LibvirtBaseConvertCommandWrapper.java 47.28% 81 Missing and 16 partials ⚠️
...rtCleanupConvertedInstanceDisksCommandWrapper.java 4.16% 23 Missing ⚠️
.../apache/cloudstack/vm/UnmanagedVMsManagerImpl.java 18.51% 20 Missing and 2 partials ⚠️
...gent/api/CleanupConvertedInstanceDisksCommand.java 0.00% 20 Missing ⚠️
.../LibvirtImportConvertedInstanceCommandWrapper.java 12.50% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main   #13151   +/-   ##
=========================================
  Coverage     18.94%   18.94%           
- Complexity    18363    18366    +3     
=========================================
  Files          6192     6195    +3     
  Lines        556361   556451   +90     
  Branches      67908    67916    +8     
=========================================
+ Hits         105397   105421   +24     
- Misses       439393   439455   +62     
- Partials      11571    11575    +4     
Flag Coverage Δ
uitests 3.51% <ø> (ø)
unittests 20.15% <35.74%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17829

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16076)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 4252 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13151-t16076-kvm-ol8.zip
Smoke tests completed. 9 look OK, 142 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
runTest Error 0.00 test_2fa.py
runTest Error 0.00 test_account_access.py
runTest Error 0.00 test_accounts.py
runTest Error 0.00 test_affinity_groups_projects.py
runTest Error 0.00 test_affinity_groups.py
runTest Error 0.00 test_annotations.py
runTest Error 0.00 test_async_job.py
runTest Error 0.00 test_attach_multiple_volumes.py
runTest Error 0.00 test_backup_recovery_dummy.py
runTest Error 0.00 test_backup_recovery_nas.py
runTest Error 0.00 test_backup_recovery_veeam.py
runTest Error 0.00 test_certauthority_root.py
runTest Error 0.00 test_cluster_drs.py
runTest Error 0.00 test_console_endpoint.py
runTest Error 0.00 test_create_list_domain_account_project.py
runTest Error 0.00 test_create_network.py
runTest Error 0.00 test_deploy_vgpu_enabled_vm.py
runTest Error 0.00 test_deploy_virtio_scsi_vm.py
runTest Error 0.00 test_deploy_vm_extra_config_data.py
runTest Error 0.00 test_deploy_vm_iso.py
runTest Error 0.00 test_deploy_vm_iso_uefi.py
runTest Error 0.00 test_deploy_vm_root_resize.py
runTest Error 0.00 test_deploy_vms_in_parallel.py
runTest Error 0.00 test_deploy_vms_with_varied_deploymentplanners.py
runTest Error 0.00 test_deploy_vm_with_userdata.py
runTest Error 0.00 test_diagnostics.py
runTest Error 0.00 test_direct_download.py
runTest Error 0.00 test_disk_offerings.py
runTest Error 0.00 test_disk_provisioning_types.py
runTest Error 0.00 test_domain_disk_offerings.py
runTest Error 0.00 test_domain_network_offerings.py
runTest Error 0.00 test_domain_service_offerings.py
runTest Error 0.00 test_domain_vpc_offerings.py
runTest Error 0.00 test_enable_account_settings_for_domain.py
runTest Error 0.00 test_events_resource.py
runTest Error 0.00 test_extension_custom_action_lifecycle.py
runTest Error 0.00 test_extension_custom.py
runTest Error 0.00 test_extension_lifecycle.py
runTest Error 0.00 test_gateway_on_shared_networks.py
runTest Error 0.00 test_global_acls.py
runTest Error 0.00 test_global_settings.py
runTest Error 0.00 test_guest_os.py
runTest Error 0.00 test_guest_vlan_range.py
runTest Error 0.00 test_host_control_state.py
runTest Error 0.00 test_hostha_simulator.py
runTest Error 0.00 test_host_ping.py
runTest Error 0.00 test_image_store_object_migration.py
runTest Error 0.00 test_import_unmanage_volumes.py
runTest Error 0.00 test_internal_lb.py
runTest Error 0.00 test_ipv4_routing.py
runTest Error 0.00 test_ipv6_infra.py
runTest Error 0.00 test_iso.py
runTest Error 0.00 test_kubernetes_clusters.py
runTest Error 0.00 test_kubernetes_supported_versions.py
runTest Error 0.00 test_list_accounts.py
runTest Error 0.00 test_list_disk_offerings.py
runTest Error 0.00 test_list_domains.py
runTest Error 0.00 test_list_hosts.py
runTest Error 0.00 test_login.py
runTest Error 0.00 test_list_ids_parameter.py
runTest Error 0.00 test_list_service_offerings.py
runTest Error 0.00 test_list_storage_pools.py
runTest Error 0.00 test_list_volumes.py
runTest Error 0.00 test_loadbalance.py
runTest Error 0.00 test_metrics_api.py
runTest Error 0.00 test_migration.py
runTest Error 0.00 test_ms_maintenance_and_safe_shutdown.py
runTest Error 0.00 test_multipleips_per_nic.py
runTest Error 0.00 test_nested_virtualization.py
runTest Error 0.00 test_network_acl.py
runTest Error 0.00 test_network_ipv6.py
runTest Error 0.00 test_network_permissions.py
runTest Error 0.00 test_network.py
runTest Error 0.00 test_public_ip_range.py
runTest Error 0.00 test_nic_adapter_type.py
runTest Error 0.00 test_nic.py
runTest Error 0.00 test_non_contigiousvlan.py
runTest Error 0.00 test_nonstrict_affinity_group.py
runTest Error 0.00 test_outofbandmanagement_nestedplugin.py
runTest Error 0.00 test_outofbandmanagement.py
runTest Error 0.00 test_over_provisioning.py
runTest Error 0.00 test_password_server.py
runTest Error 0.00 test_persistent_network.py
runTest Error 0.00 test_portable_publicip.py
runTest Error 0.00 test_portforwardingrules.py
runTest Error 0.00 test_primary_storage.py
runTest Error 0.00 test_primary_storage_scope.py
runTest Error 0.00 test_privategw_acl_ovs_gre.py
runTest Error 0.00 test_privategw_acl.py
runTest Error 0.00 test_projects.py
runTest Error 0.00 test_purge_expunged_vms.py
runTest Error 0.00 test_pvlan.py
runTest Error 0.00 test_quarantined_ips.py
runTest Error 0.00 test_regions.py
runTest Error 0.00 test_register_userdata.py
runTest Error 0.00 test_reset_configuration_settings.py
runTest Error 0.00 test_reset_vm_on_reboot.py
runTest Error 0.00 test_resource_accounting.py
runTest Error 0.00 test_resource_detail.py
runTest Error 0.00 test_resource_names.py
runTest Error 0.00 test_restore_vm.py
runTest Error 0.00 test_router_dhcphosts.py
runTest Error 0.00 test_router_dns.py
runTest Error 0.00 test_router_dnsservice.py
runTest Error 0.00 test_routers_iptables_default_policy.py
runTest Error 0.00 test_routers_network_ops.py
runTest Error 0.00 test_routers.py
runTest Error 0.00 test_scale_vm.py
runTest Error 0.00 test_secondary_storage.py
runTest Error 0.00 test_service_offerings.py
runTest Error 0.00 test_set_sourcenat.py
runTest Error 0.00 test_sharedfs_lifecycle.py
runTest Error 0.00 test_snapshots.py
runTest Error 0.00 test_ssl_offloading.py
runTest Error 0.00 test_ssvm.py
runTest Error 0.00 test_storage_policy.py
runTest Error 0.00 test_systemvm_userdata.py
runTest Error 0.00 test_templates.py
runTest Error 0.00 test_update_security_group.py
runTest Error 0.00 test_usage_events.py
runTest Error 0.00 test_usage.py
runTest Error 0.00 test_vm_autoscaling.py
runTest Error 0.00 test_vm_deployment_planner.py
runTest Error 0.00 test_vm_life_cycle.py
runTest Error 0.00 test_vm_lifecycle_unmanage_import.py
runTest Error 0.00 test_vm_lifecycle_unmanage_kvm_import.py
runTest Error 0.00 test_vm_lifecycle_with_snapshot_or_volume.py
runTest Error 0.00 test_vm_schedule.py
runTest Error 0.00 test_vm_snapshot_kvm.py
runTest Error 0.00 test_vm_snapshots.py
runTest Error 0.00 test_vm_strict_host_tags.py
runTest Error 0.00 test_vnf_templates.py
runTest Error 0.00 test_volumes.py
runTest Error 0.00 test_vpc_conserve_mode.py
runTest Error 0.00 test_vpc_ipv6.py
runTest Error 0.00 test_vpc_redundant.py
runTest Error 0.00 test_vpc_router_nics.py
runTest Error 0.00 test_vpc_vpn.py
runTest Error 0.00 test_webhook_delivery.py
runTest Error 0.00 test_webhook_lifecycle.py
runTest Error 0.00 test_host_maintenance.py
runTest Error 0.00 test_hostha_kvm.py

@sureshanaparti sureshanaparti requested a review from Copilot May 13, 2026 07:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a best-effort cleanup path for temporary converted volumes when VMware→KVM import fails, and refactors KVM conversion wrappers to share common helper logic.

Changes:

  • Trigger cleanup of temporary converted disks on import failures/timeouts.
  • Introduce CleanupConvertedInstanceDisksCommand and a corresponding KVM resource wrapper to delete leftover conversion artifacts.
  • Refactor LibvirtImportConvertedInstanceCommandWrapper to reuse a new shared LibvirtBaseConvertCommandWrapper.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java Sends a new cleanup command to remove temporary converted disks when import fails.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtImportConvertedInstanceCommandWrapper.java Refactors wrapper to inherit shared convert/import helpers and updates cleanup call signature.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtCleanupConvertedInstanceDisksCommandWrapper.java Adds KVM-side handler that locates and deletes temporary conversion disks (and XML when present).
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtBaseConvertCommandWrapper.java Introduces shared helper methods previously embedded in the import wrapper.
core/src/main/java/com/cloud/agent/api/CleanupConvertedInstanceDisksCommand.java Adds agent command to request cleanup of converted disks by store + prefix.
core/src/main/java/com/cloud/agent/api/CleanupConvertedInstanceDisksAnswer.java Adds a new Answer type for the cleanup command (currently empty).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread core/src/main/java/com/cloud/agent/api/CleanupConvertedInstanceDisksAnswer.java Outdated
Comment thread server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java Outdated
@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@harikrishna-patnala harikrishna-patnala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code LGTM. @nvazquez please check if we can add some unit tests here.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17841

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18334

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16395)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52785 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13151-t16395-kvm-ol8.zip
Smoke tests completed. 150 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestClusterDRS>:setup Error 0.00 test_cluster_drs.py

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18356

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18367

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18370

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16428)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50929 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13151-t16428-kvm-ol8.zip
Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@weizhouapache

Copy link
Copy Markdown
Member

@nvazquez
is this applicable for 4.22 too ?

@weizhouapache

weizhouapache commented Jul 8, 2026

Copy link
Copy Markdown
Member

@nvazquez
it looks like this PR can be applied to 4.22 too.

(4.22)# patch -p1 <13151.diff 
patching file core/src/main/java/com/cloud/agent/api/CleanupConvertedInstanceDisksCommand.java
patching file plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtBaseConvertCommandWrapper.java
patching file plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtCleanupConvertedInstanceDisksCommandWrapper.java
patching file plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtImportConvertedInstanceCommandWrapper.java
patching file plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtImportConvertedInstanceCommandWrapperTest.java
patching file server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java
Hunk #2 succeeded at 1721 (offset 3 lines).
Hunk #3 succeeded at 2219 (offset 3 lines).

Could you rebase with 4.22 ?

I am moving it to 4.22.2 milestone

@weizhouapache weizhouapache modified the milestones: 4.23.0, 4.22.2 Jul 8, 2026

@RosiKyu RosiKyu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Executed the full test plan for FR307: VMWare-2-KVM part II - Cleanup after failure - 19/19 test cases Pass (100%).

Summary:
Testing covered cleanup of temporary artifacts (disks and XML descriptors) after failed VMware-to-KVM imports: temporary locations on the primary pool and NFS secondary storage, XML handling (present / absent / corrupt with fallback to prefix listing), forceConvertToPool positive and negative paths, mid-move multi-disk failure (customer repro), destination-pool orphan check, and correct cleanup scoping (concurrent imports and the convert-stage boundary remain untouched). Negative scenarios confirmed no crashes when there is nothing to clean or when cleanup itself fails, with the original error preserved. Failed conversions are correctly reflected in the UI and in listImportVmTasks, and the successful import regression passed with temporary artifacts fully cleaned.

Executed test cases:

  • Concurrent imports cleanup scoped to failed prefix only
  • Import fails after temporary disks already cleaned nothing to clean no crash (negative)
  • Convert host equals import host VDDK same host cleanup works
  • Import VM tasks UI reflects failed conversion
  • listImportVmTasks shows Failed record on pool selection failure
  • Convert-stage failure not cleaned scope boundary
  • forceConvertToPool success no erroneous cleanup (positive)
  • forceConvertToPool import failure cleans pool disks (negative)
  • Successful import regression temporary fully cleaned (positive)
  • Temporary location primary pool cleaned
  • Temporary location NFS secondary cleaned (positive)
  • Destination-pool orphan check after mid-move failure (negative)
  • Mid-move multi-disk failure temporary fully cleaned customer repro
  • Cleanup with corrupt XML falls back to prefix listing (CS-3416)
  • Cleanup with XML absent removes disks skips XML
  • Cleanup with XML present removes disks and XML (positive)
  • Import failure agent unavailable cleans temporary disks
  • Cleanup failure preserves original error and does not crash (negative)
  • Import failure bad result cleans temporary disks (positive)

@weizhouapache

Copy link
Copy Markdown
Member

thanks @RosiKyu

discussed with @nvazquez , merging into main/4.23

@weizhouapache weizhouapache merged commit 72197fe into apache:main Jul 8, 2026
26 of 27 checks passed
@weizhouapache weizhouapache deleted the 423-vmw-kvm-cleanup-migrated-disks-failure branch July 8, 2026 11:07
@weizhouapache weizhouapache modified the milestones: 4.22.2, 4.23.0 Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants